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IDENTIFIERS 
ABSTRACT 

The intent of this paper is to illustrate the use of 
stakeholder information in evaluating a school program. The material 
presented is part of a comprehensive formative evaluation of a .crisis 
intervention program operated by a suburban school distric't situated 
near a large industrial city in the Midwest.. The crisis in^rvention 
program provided counseling serviced to high school students _ 
experiencing personal problems, such as substance abuse, parental 
drinking, sexual abuse, grief and loss, and depression and suicide. 
The project illustrates the feasibility of a "triangulation of 
evidence" technique in which behavioral information from case records 
is used to supplemen't interview and questionnaire responses of 
various stakeholder groups. The term stakeholders as ustfd here > 
specifically includes the recipients of the service as well as third 
party groups impacted by the program. The illustrative study. is 
followed by a discussion of the advantages and disadvantages of the 
present approach, and considerations regarding replication. The 
appendix is comprised of nimerous tables dealing with choice of 
professional to intervene with various problems. (BW) 
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I. INTRODUCTIOM 



The intent o£^ this pap. r Us to illustrate the use of f ' 
stakeholder information in evil^it^^ program. t»t . 

mate riVl presented is par t of M ^ format^fve^valuati^n 



of a 'crisis interment 



i on 'Vprogr» 



operated a afuburbah school 
district si tuated near * Urge \i)nd« tKa. Mi 4w5 1 . ^ The 

proiect Il lustrates a " tr ianqulU ioti o£ evideW' technique in >.hich 
behavioral inf ormat ion f rpiik casi records are us,d to supelem«nt , 
interview and qtiistionna^re resiLnses of various s tafceho Ider gf oup^^ 
The term stakeholders as used heVe spec i fica l.ly: inc ludes the *|v 
recipients of the service rs well as third party groups i^^^^^^ 
the program. - ThV illustrative stlidy is Tollowe^ by a discussipn oi 
the advantages and dl»<^dyantages if t>e present approach, and- t 



considerations regarding replicatilon, ' - ') 



Over the past 20 years/ the r.<ile of stakeholders ("i.e. those 
individuals and groups with vested Uteresti in a program ) has 
become ihc;reMirtflly P^ominant In educational 
evaluation--particularly formate e.^^ Mist of the 



eval5kt ion jSodtIs which have em«tged sincft, the late l^40;.s ^ ^ ^^ ^ ^ 

-i. . ■ -. . ■■ ■ . -/ ^ ■ ■ '" 'i' ■ '■■ 

Metfessel jand Michael , 1*^7 ; Sjkuf f lebeam, et al . , tX^7 1 ; Kllcin, 
Boxich, . tf74> call for stakeholder input . More recent ly , several ; 
eva luato:r^ cliose to build the i r *app roaches around stakeholder input 
(e..g. Bryfc, 1983). Despite^his universal concern, however,^ 
comprehensive strategies for ut i f i i ing st akeho 1 der ^r^pu t have yet to 
be fully ^ deve loped . None of the authors cited provide specif ic 
me^thods .*that appLy across settings. Suggestions which do appear in 
the literature concent'rate on verbal input from gi»Ven^ gro'up mesAers 
(e . g. 'interviews , questionnaires, use of weighted scales, panel 
discussion^, etc.; for detailed presenta,t ions of such methods, refer 
to Patton, 1980, and Edwards, et al, 1975). The verb* T r epor t duta 
Obtained in most stakeholder basFed /evaluations appear to (ocus on 
the following issues : "Vhai quest ions^ are the most useful to ask 

about the outcome? To whom* are the answers important?'^ 

' /' ', ■ ^ 

■ * \ ■ ' • - • ' . 

Asking ^stakeholders' to answer questions such as these has 

■ * , .■ 

several inherent advantages, including the ident i f i,cat ion of 

relevant measures to-tise;' the identification of preferred mfans of 

•' ■ ^ . . <^ ■ 

present ing-? r esu 1 ts , ^nd prompting the interest a*nd par t icipat ion of 
stakeholders once results are presented (Stake , 4983) . Howejrer, 
several issues remain unaddressed. One sOch issue is what Collet 
calls "goal distort ionV (Col let , Wa 1 « / and Co 1 le t", i *83 ; Collet and 
Pi'ntrfch, 1980; Collet, 1^77): what par ticipailts- say about the 
effectiveness of > program does not necessar i ly ref tect it's trde 
effectiveness — it may only reflect i ts ability; to pi ease 6r 
displease. Using verbal input as tH« «ole basis for ^fjo'cmulating 



evaluation measures can. serio^usiy* distort the foci atTui results of a: 




evaluati?fi^( '(for a case in ' pp int , see Farrar.and H6tJtf'%;:'l9ff3 ) . ' 

al though , the strategy of assess ing ver balv input can'' 
n^xfy the relevant evaluation issues aiid^agr^^ments and 
with regard to/ those issues/ it neither clar if les th,e 
sagreemeht I nor identifies the consequences ' of those 
disagreenents for- the op^ecation' of the program. 



[mary purpose of thks p^per . is to illustrates the 
f easibi 1 i^^^of using both verbal and . behav i of a 1 measur es^ in the 
assessment of stakeholder" input . Measure? of .actual stakeholder 
behavior can be. ob t a ined f r om seve r al sources /includ ing existing 
documents - (e . g . case r eccfxds , school recoxds) , evaluator 
observation, pa r t i c i pant oi^erva t i on , and participant 
self-monitoring (e.g. daily checklists). .^ ^ 

The inclusion of. behavioral measures with verbal reports, and 



the subsequent tr iangulat i on of ev idence <;o"uld strengthen 
stakeholder based eva luat ion in the fol Towing ways^ 

■r ■ i . ■ ■ ■ « ■ ' - - . 

Clarify the Validity of a priori groupings of 
stakehol ders . 

Quality and speci f ici ty of f eedbacip to participants, 
could ^be enhanced. 

Global agreements and /or di'sagreements among ^ 
stakeholders could be clarif ie5 and focused by 
'examiiving actual behavior. 

Behavioral measures of stakeholder involvment could 
help i so late or pare down the factors in the program^ • 
environment influencing verbal ratings, descriptions, 
aAd positions (e . g . are^ di f f erences among groups 
influenced by unequal payo f fs for respective members, 



Jft 



by4 dif f eTftncA's in access to inf ormat ion« by ^ ' 
constraints inherent in group members' roIeSi by 
physical [^access to facili ties, etc.). 

Verbal pos i t i ons , * i I I.umina ted by behavioral measures, • 
can help isolate speci fie points in the ecology of the 
program that ian be target ted f or clar i^ icat-ion (i.e. 
more st^fceKo I det input >, inter vent i on and modification 
(e.g. is the most ^appropriate intervention a change 
in administrative policy, add i t ional inf^rmat ion and 
\ training f qr . cer t ain groups , ar r angi ng comp 1 iance 
ince^nt iv.es, etc.). 

• ' ■ f ^ 



II . THE PROJECT 



The Program 



^ Between mid-1979 , and September of 1982, parents and sclj^ool 
officials identified several problem areas that affected the teenage 
population in the community. Thifse included substance abuse, 
pare'ntal^r inking , sexual abuse, grief and loss, and depression and 
suicide. The r ecogni t i on of these problems presented several 
challenges to the school systems and the community. First, such 
personal problems were seen as barriers to learninii, Academic 
achievement, soc i aT ad}us tment , and long-range ^goal attainment. 
Second, although the problems had a direct impact on^^school ^ 
per f ormance ^and adjustment, there was.no comprehensi've strategy to 
deal with them wi thi n the school. This "service gap" Prompted the 
school board to initiate a crisis intervtion program for ^h4 
1982-1983 . school year in the district's two senior high schools, 



which had a aombinjftd enrollment of som« 3il50. students 



. The crisis i nt er vei>t i on pr ogi^aju^as intended^Jo provide several 
educational and^ counsel'ing se rv i ceir on ,a number of levels. These 
inc luded inser.vice training sessions f.or teacher s and 'counselors , 

consul tat ion with middle schoo'l Counselors , spec ia 1 -top i c small 

' . • ■ • ' . • - > ■ , - . 

group sessions wi^li students , -d i rec t^ counsel ing services to student^ 

in crisis and their famil ies, and referral to community resources K^ 

whenever appropriate. ' ' - - , 

To implement" this program, two intervention counselors were 

«• ' ■ 

appointed. Their role w^s largely personal counse 1 ing with students 

*• *' ' _ ' . 

( 6 time) and consul tat ion Wi th teachers and general counselors (. 2 
time)'. The appointments r^cogni led the need for eKpanding services 
to hlah risk youth , wh i 1 e maintaining a consistent level of guidance 
and counseling services provided by the regular c^>un5eling staff. 
Other prof esS'ionais from the community wfre invited to cpnduct the 
inservice sessions-, ^ \ 




B. Mithods - % ' 

, * ■ -. ■ • ' ^ ' 

• "• ■ » ■ •' * ' 

, The cr isi s t counse 1 ing program was deve loped to deal«wi#h 

Personal problems of *n jiraent natur*. It was th«rfefore cons ide r eid 

unethical to use the usual control or de la^ed t r ea tment coji^par i,son - 

groups f or evaluat ion. To at least par t ial ly compenste» for this , 

weaknessi the eva lua t i on' was designed to f aci 1 i tate tr^angulat ioii *of 



evidence. The term ""^t riangu t.i on^ comes from a ge*ometric principle 
used by 'land surveyers to precisely Idycate the posit i(yi and 

■ y ^ ■ . : / • m!^ . . • . ■ ■ ■ ' . 

elevatioji of a distant object* such as a-Vmountain top. As used in 

evaluation (cf. Collet, Val^ and Collet, l?83, p. 21-22), 

' • ' ^ 

tr iangulation* ref f i^s to the systnemaj^ic collection of data from at 
least three * di f f erf nt perspectives : ^ per cep t i ons o f stakeholders, ^ 

detailed program. records . g . > patterns of servLce utilisation), 

/f^ ' ^''^ . ' • 

and recoxds of client changes over time. 



The overall V^a lu^t iib'^^ tf>ree dist inct i|^nDonent« : 



stakeholder assessment, evaluation d f inser v i.t:e sessions # and the 

examihation of problems dealt with and case d iiposi t ion . For the 

. ^ ' ' ' ^- ' 

stakeholder portion of the evi^luat ion, data were collected by three 

' *' . • ^' ■ . ■ * , ■ , ■ ^ . . ' ■ • ' 

means; First, interviewii ;were conducted with the crisis counselors 

: ^ ■ . / ■ . . * ■■ 

themse 1 ves and with the program* s developers^ the director of the 

• • * ■ 

district's office of guidance/and :counseling, and repr sentat i ves 
from the school board and from^ community action group. Second, a 
set of anonymous ques t ionnai rjts was designed to gather input from 
parents, students, and ins t r uc t i ona^l stafj — .groups too large and 
div^erse to effectively us^ inter viewing " as a method of gathering* 
representative data. Third, measures of stakeholder behavior were" 
coded from the case records ofjrthe crisis counselors, along with 
other information. 



Interviews were conducted to define an number of evaluation 
issues, including the following: how .is a cr i s i s counse to r 
differentiated from a general counselor , what kinds of Ajfit comes 



would be indicative of program e£ feet iveness; and what kinds of 
other issues are v important with regard to program imp-Iementat i on . 
Another purpose o£ the interviews was to generate items for use x>n 
the questionnaire. ^ 

V ■ 

■ 4.r ■ J 
ffach sftt of qu«stiorttt»ir*« had two sections. The first 

consisted of & list , of tKorolilems likely to be encountered by high 

school students; these incl\ided 11 personal pi;oblems like depression 

and substance abuse, #nd. 10 "guidance" problems like schedule ^ 

-changes and choosing a college. Participants were asked,!© choose 

which school professional , if , any, was most .appropr iatte to deal with 

that problem. The second section was comprised of L^kert scaled 

items referring to the overall appropriateness and desirability of a 

crisis intervention t>togram in a public schoo 1 se t t ing . The staff 

version of the questionnaire also provided space for •open ended 

-comments. Because of th« re la t i ve 1 y gr eat er , number of respondents, 

student and parent versions did not have this feature. The student 

version also had an additional section t o be f i 1 1 ed ou t by those 

individuals who had received services from the crisis counselor. 

Consistent with the requests of school board members, 
questionnaires were distributed as a census to some 3,150 students, 
and 150 instructional staff members. Als.o, questionnaires were 
mailed to 1600 households to. gather iYiput from parents. Because of 
the niture of 'the sampling, only univariate and bivariate frequency 
distributions were used to eKamine results. 



E'xisting case records wer^ the source of the behavioral data. 
V The crisis .counselors were taught to use a codebook to extract* 

specific informat ion from . the records and tranfer it to a computer 
readable format. Each case was coded for the fo I lowing ^ informat ion: 
sex of client/ curriculum enrolled in« source^^f ref'erral, 
presenting problem at, referral, problems dealt with in^counsel ing 
(Locus in treatment) types ol services rendered, and out*come 'of ^ 
cases ( case . d i spos i t i on) . Because the counselors themselves 
invented a new idei^ti f i cation number^ for each^caswei they v^ere the 
only ones who could match the computer codes with the actual cases. 
In this manner, coifnse^or/cl ient confidentiality was preserved. 

% . . '■ ' . ' ■ , ■. ■ 

i' . . . 

C. Results: Verbal Report Data 

Interviews with program developers and the. crisis counselor's 
themselves revealed that aside from bei4ig interested in *inf brmat i oh 

V on actual problem incidence and case outcomes, these individuals 
were most concerned about o ther s t akeho I der s>^ perceptions of^^and 
pr i or i t i es f or the program, because, ^for the most part,, these groups 
did not participate directly in the development of the p^rogram. The 
issues considered most important are listed as follows: should 
personal problems of students be dealt with in the school; which 

\ particular problems, if any, are considered appropriate for 
irnt er vent i on; who is seen as the. most appropriate schodi 

'professional to infervene; is a formal iSrogram that deals with 

< 

students*^ personal problems appropriate in a public school setting;^ 

in ^ 



should such a program be eEpand#d to serve a younger (i.e. middle 
school) population; and are there ^ny concerns over the fact that 

both crisis counselor-s are male . Subsequently, these is-sues 

* • ' ' - 

comprised much of the contren^ of the questionnaires directed at 

y ■ • ' ^ . * ^ 

parents, students, and/ instruct ional staff. 

X. total of 1,978 students, 445 patents, and 83 members of the 

...... » 

instruct ional* staff responded to questionnaires. With* few 
exceptions, responses to Lifcer.t-scaled items i^ppea'red to be 
consistent across all three groups, and across schoo Is . A majority 
of alrl responden»ts agreed that a <irisis counselor should be 
available^in the /d i s t r i c t ' s two senior high schools <?2.4%, 82.3%, 
and 73.9% of parents, staff, and students , respect ively) . A 
majority of each'of the three stakeholder groups disagreed with * 
items which stated that students' personal problems should not be 
d«alt with in Vlr« school. A majority of both parent* and students 
agreed that they needed more information about the current program. 
While a majority o| pareitts did, riot object to a student being seen 
by'a counselor of the opposite sex <65.9% regarding female students 

t 

.«%|ing a male counselor, 'and \67 . 7% regarding the reverse situation) 
students split oh tii^s issue <36.6% agreed that they would be 
uncomfortable. 3?. 8% disagreed, and 23:6% expressed "no <«>inion">; 
these distributions remained consistant when stratified by sei of 
.the respondent. 

Some items were directed at speti f ic stafceho Ider groups. For 
example. 80 . 3* pf the parents agreed, t^hat su^h a program should al! 



bft available £o r youngtr students . On a. set of special i sed it^ms, 

. . • : Q ' ■ ■ ^ ■. .• * 

86.4% of staff members agre^ed'-Jthat theyiWere comfortable ifliferring 

students to> the crisis counselori while 71.1% of this group 

disagreed that they hesitated Jto consul t h im. , • ' / 



In general « pareT^ts and students tended to agree wi th each 

■ ■ ■■■■•v.;' ■ ■ ■■■ ' "-^ : ^ . 

vther, over which sc^h»ooI professional was most appropriate to 
intervene with gi ven* prob I ems . These^ respondent s agrtie<f tiat \ .r- 

planning for the nekt term« schedule changes « -chooring f college;^ 

< r . 'v. . ' ' . * I 

'■ ' * . . • 

school per f orAance , f ai I ing an e«aiai na t ion, flunking a semester r and 
adjustment to a new school were wi th in the genera 1 ^ounselo r sV / v 
domain. Parents^and students also selected the crisis 66unseIoV to 

' * ^ ■ ' ■ ■ r ■ V *■ ■ (^'" (• • 'a 

be the mos t app r opr i art e ' p r6f ess i onal to fdeal wi th drp r ess i oh , ^ /* 
suicide, alcohol abu'se'i other subs trance abuse , physical abuse, and 
pregnancy. However, students fqund iexiully t ransihi t ted di seases, 
sexual abuse (i.e. in the^ome), grief and loss i ar res t , and 
breafcijig up with a boyfriend or g i r-lrfr i end not appropriate for 
intervention l"n the school. Parents found only the breakup i*sue to 

■»■ ■' a, 

be inappropriate, while attributing the o ther si t i ons\ to be the 
crisis counselors' responsibility. L ike responses on the 
Likert-scaled items, the distr ibut ions of responses on these it'ems' 
were .consistent across schools. , ^ _ 



Consistent with parent and student responses ,. there appeared to 

be a strong consensus amon^g ^ teachers wi th regard' to the* role of^ thte 

■ ■ - « . ' 

general Counselor. A majority of staff members txom both schools 
chose this professional as the most appropriate to intervene with 
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the foMowing problems:" planning for the neKt term, schedule 
changes, choosing a college, flunking a semester, failing an exam, 
school performance, and ad j us tment to a new school. 



The ^staf f .responses regarding the most appropri.ate professional 
to intervene with students' per sona 1" p r ob 1 ems appeared to differ by 
school (see App^endix, Tab les l-l 0 K^^^r ob 1 ems concerning grief anH8 
loss, physical abuse/ sexual abuse, depr ess i on/ su ic i de , a Icoho 1 
abuse, other subs t ance abuse , and pregnancy were seen as being in 
the crisis counselor's domain by a majority of staff in one school 
(School B) ; for the same items, the op ini ogris of the staff of the 
other school (School A) were split between the cr i sis counse 1 or and 
"either the crisis counselor or the general counselor" (see 
Appendix, Tables 2-8). With regard to sexually transmitted 
diseases, the consensus at School A was split between the crisis 
counselor and "either counselor,^' while at School B the split was 
between the crisis counselor and "another^school prof essional" 
(presumably the school nurse) ( Append ix , Tab 1 e ?). F ina 1 ly , wh i le 
47:4% of School A' s staff chose the general coun^elNor^to be the most 
appropriate professional to deal with school attehdance, "another 
school professional" (presumably the attendance officer) was 
designated this way by' 63.6% of School B's staff (Append it Tab le 
10). A similar split was noted with regard to the problem of 
skipping classes. - 

(I . 

While the numbers of parents and students were too large to 
request *addi tional remarks, comments were so 1 ic i t ed f om s taf f 



members at the end of their version of the, q^es t i onna i r e . Of 83 

staff members re turning questionnaires, 50 chose to include 

•> • , * ■> 

ad^^tional comment s . ^ Vi th rega^rd to the counseling department; 



respondents almost uni versa I ly I i s ted **coord ina^t i on of information" 
as an afea the neieded change. More specific comments in this ar;ea 
included the fol lowing : the neftd for mortf teacher /counselor contact 



regard ing students * academic concerns .and choices; the issue of 
counselors^ 'informing teachers about "student^s* spec iai needs" and 

giving specific reasons for removing students from class; the issue 

■ ' ' ■ • ■ • 

of rcM^oving students too f requent ly from the same class for 

cojifer ending ; and the issue. of inc lud ing counse lor s or 

"^^'v,' ■ . - , . - . . ■ ■ ■ 

counselor-representatives at teachers* staff meetings to facilitate 

ongoing dialogue. 



Although remarks were no t request ed « 45 parent questionnaires 
v/ere returned with comml^^ts. Most, of these comments emphasised the 
need for st r ictest conf ident ial i ty for students with personal 
problems. Other comments, ral sed th^ issues of parental consent for 
personal couns^eting, and the need for parental participation in the 
^counseling process. . ' ^ • 

Finally^ each student questionnaire included a section to be 
filled out by the recipients of counseling ^erv i ces . However , 
because the number of r^esponde^nts here greatly outnumbered the 
number of actual case records, these results will not be reported or 
. interpreted . 



To sumkariii, the di aagreame^Wi in stakeholders' verbal report 
data occurred between par^^ and students with regard to the 
appropiateness or some speci f ic problems £|^r school intervention, ^ 
and between parents and staff with regard to confidentiality. ,The 
most consistent .between-school disagreements were among st^M>^with. 
regard to the^ "most appropriate professional to intervene" with 
students * personal problems . 



D. Results: * Beha^idral Data 



A total of 376 cases (11.7% of the ehtire enrollment) seen by 
the counselors on one or m6re occasions between September 1, 1?82, 
and Aprif li 1983 , were eKamined by coding specific information from 
the case records into a computer readable format. Behafioral daU 
from thj^s« case records revealed more extensive be tween-schoo 1 
differences. Most of the cases at both schools were self-referred 
(47.1* for both archools combined). However , whi le the majority of 
the cases at School A were either se If -re f er r ed , or referred by an 
administrator (58* and 20.7*. respect ively) . the cases at School B 
were referred by a more diverse cross-section of school 
professionals and other, individuals, wi th 35 . 5* be ing se 1 f -re f er r ed 
(see Appendis, Table 11) . 

The most common types of service delivered were 'individual • 
counseling, and individual counseling accompanied by either a 
concurrent or subsequent referral (33.4* and 22.?*, respectively). 



Howttveri th^e appeared to be be tween^schoo I d i f ferences . in the ^ 
patterns of service delivery. kt School k, individual counseling 
was : the service provided most f r equen t ly^i « wh i le the nest most 
commonly provided service was group counsel ing (49^ and 30.2^i 
respectively). At School B, howeveri individual counseling with 
referral was the most frequently provided service (-dS^S^)! followed 
by inhouse referrals ^^^^ i s « * r e f erra I s to other professionals 
within the building ( 2 6 . 8%^ ( see Append i x « Table 12> . 



Discussion of Results 



Differences in verbal report , particularly of staff members, 
paralleled differences in actual behavior in the context of the 
program. Staff members in School ^A did not seem to differentiate 
among school p fo f ess i ona 1 s with regard to the "most appropriate 
professional to intervene*' wi th students ' personal problems, while 
staffers at School B, lik« parents and students, appeared to draw 
sharper distinctions among the roles of school' personnel . 
Paralleling these \d i f f erences were contrasting patterns of referral 
and service delivery in the two r espec t i ve schoo 1 s . Referrals ^t 
School B tended to come from a var iety of school pro^f ess i ona 1 s and 
other individuals, whi,le referrals at School A came predominent ly 
from the students themselves, and from administrators (Appendix, 
Table 11). Similar be tween-schoo 1 differences appeared in the 
diestr ibut ion of 6ases by typ» of serv i ce ( Append i x , Table 12>. 



While almost 80%k of all cases at S^hpol A were sen in individual or 
group session*, the majority School B's cases were either seen, 
injd^vdualiy with concurrent referral, or were referred inhousei 
although contrasting clinical styles can account for many of these 
difference*' they are, nonetheless, consistent with difference^ in 
referral {patterns. 

i ^ . ■ . 

\ I - ■ ' • ■ ■» • 

One explanation fo*f the above be tween-schoo 1 differences is 

"That the \iounsel ing departments of the' t^o high schools may have 
evolved different servi ce de 1 i ver? systems, with School B counselors 
developing roles which teachers perceive as spec i al i red , whi le 
School A counselors may be seen as 'having more "generic" functi^bns." 
TJiiseKplanat ion can account for some of the differences in service 
delivery patterns, and In staff perceptions of crisis counselors' 
roles. However, s t a f f s 'Jc 1 ear consensus regarding the general 
counselors' roles i-n both schools seems somewhat Inconsistent with 
this explanation, and differences in referral patterns are not 
accounted for. 

Another explanation for the above patterns Is that there may be 
fewer opportunltes for communication among professional groups at 
School A than, at School B. This explantlon can account for a , 
broader diversity of referral sources and greater frequency of 
inhouse referrals at School B, as well as more distinct staff role 
differentiation at that school, not only between crisis counselors 
and general counselors, but between the counseling department in 
general and other school professionals. This explanation can 

17 



^ ; ' ■ . ■ V - . ' „ . 

accomodatA the consensus regard Ing the general counselors' role, — ^ 

* . ' ■ ■ •• 

* which is more traditional aiid better established t han , the cr is i s 
counselors'. A factor .nbvt previously cons idered \hat supports this 
eKi#lanation is the arohitlctural layout of the two high schools: at 
School B, the counselors' toff ices, teachers' lounge, and \ 
administrators* offices ark centrally located and adjacent to each 
<lther, while these faci.litlles at School X are physically seperate. 

^ It must beVnoted that thiW proximity factor is not incons4stent with 

... ^. . . • ^ ■ 

^ either the first eHplanat ion, or with the notioH ,of different 
' • ■ * > \ ' , ' 

clinical •tyle«. In fact, all thr«« s«t« of the aibove circumstance* 

may b« operating concurrent ly . Despite between-«choo 1 differences. 

the Staffs of both schools appeared to be supportive of the crisis 

counselor program. • . Wt 



The vn^h^iLl report section of this stakeholder assessment served 
some of themaror Vt^^ of conventional s takeho 1 der * based 

evaluation (cf. Weiss, 1983): it' clarified the developers' concerns 
regarding other stakeholders' perceptions of program 
appropriateness, and introduced confidentiality policy and 
intra-staff coor d inat i on ' of information as crucial issues not 
previously addressed. Furthefmore, tr iangulat ion ot verbal input 
with behavioral measures served to clarify the impact of various 
stakeholders' perceptions on the operation of the program. 

The stakeholder section of this evaluat i^n al so provided 
information which proved invaluable with regard to the clarification 
of inf ormat ion^col lected in other phases of th* evaluation. For 
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e^iampltt^ different patterns of problem incidftnce in the two schools 
were seen as partly a function of differential labeling at refe^rral, 
and not exclusively as differences either in student populations or 
in actual problems experienced per se. On the basis of the 
stakehoIde)r information, teacher information needs were redefined as 
being^most concerned with clarification of policies an-d procedures, 
and not the identification of students* personal problems, as- the-^ 
developers originally assumed when they designed inservice training 
sessions'. All the above inf orma t i on was subsequently ut i 1 i xed to 
make ch^a^ges in policy, record keeping practices Ci,e. with regard 
to problem incidence), and design of teacher inservice sessions for 



the following school year. >^ 



II I IMPLICATIONS 



The data ga the r ed conf i tmed much of what the authors expected 
to find-. iiriangulation of verbal and behavioral evidence improved 
the qualtty of feedback to s | akeho 1 ders , clarified actual 
stakeholder involvement in. the program, identified a major 

- *' ■ 

.,*-"'■' ' • • ■ 

unanticipated factor iiif I'^^n^ing prograi^ operat ion ( i . e . physical., 

setting), ^jid i dent i f 144^^^^^^^^^ Intervent ion targets (i.e. the 

need /for informat ion exchange pol icies , especial ly where 

architectural constraints 1 tml?j informal exchange). The authors* 

initial contention that verbal l||tffu^ adequately represent 

what actually goes on in the progr5i^: is demonstrated by 



th«. tact that a majority of staff a«mbers of both schools agreed 
with a statement that they 'f e 1 1 comf or tab le in referring students to 
the crisis counselor (86.4%)i while they disagreed that they t 
fiesitant to consult with him (71.1%). As illustrated previouslyj 
these statements did not refl-ect actual ^behaviori as teacher 
referrals were virtually non-existant in one school. . 



Ifiis 



'The major problem encountered throughout tliis project was ^ 
non-compliance. Only 62% of the students. 5S% of the instructional 
staf {^i and 2a% of the parents contacted ch<^se to respi^y^^d-- to 
questionnaires. The developers' i ns is t ance that the schools 
distribute and collect the questionnaires may have had an impact on 

this low r a't e . Questionnaires for teachers and students were 

"... * ^ 

administered over a one week period in a variety of settings (e.g. 
faculty lounges, study halls, homerooms). Furthermote« because of 
the sampling method used (census) and^the lack of specific 
identifying information on the quest ibnnaies (i.e. to insure 
confidentiality, as school personnel were handling the materials), 
follow-up procedures with regard to verbal input were precluded. 

The findings here lend strong support for the notion of 
incor pora t ing behav i ora 1 measures wi th ve r ba 1 report in as s*ess ing 
stakeholder input . Replication under a variety 6f conditions is 
necessary to establish the present approach as a viab leeval uat i on 
technique 'across programs and settings. However, the present 
example of ' incopora tng ^strategies more of ten discussed wi th regard 
Xo "implementation evaluation" (Stuff lebeam, et al . , 1?71), and 



0 



,"pcoc«ss evaluation" (Patton, 1?80> into stakeholder b^ed . 

0 . 

approaches can «erve to strengthen the latter approach and help 
evolve its usefulness. 



J 
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TabU 1 
Pareant of Staff Choicaa 
Moat Approprlata Profalalonal 
To Intarvana with Family confwcta 





i 

School 


Moat Approprlata Profaaalonal 




Total wlthln- 
Counaal ing 
Dapartmtfnt 


Criala 
Caunaalor 

■ • 


Qanaral 

Counaalor 


i 

Elthar 
Counaalor 


Anothar 
School 
Profaaalonal 


Not 
Approprlata 

For 

Intarvantlon 

in School 


At 

SCHOOL A 


15.4 


23. 1 


38.5 


7.7 


15.4 


y 76.9 


At 

SCHOOL B 


29.4 


^ 17.6 


, 38.2 


2.9 


11.8 


85.3 


TOTAL 

Both School a 


21.9 


.'20.5 


38.4 


5.5 


13.7 


80.8^ 



Tabla 2 
' Parcant of Staff Choicaa 
Moat Approprlata Profaaalonal 
To Intarvana with Qr laf and Loaa 







Moat Approprlata 


Profaaalonal 






Scho^ ^ 


, Criala 
Counaa 1 or 


r 

Qanaral 

Counaalor 


Elthar 
Counaalor 


. Anothar 
School 
Rrofaaalonal 


Not 
Appropriate 

For 

flintarvantlon 
In school 


Total within 
Counaa 1 Ing 
Oapartmant* 


At 

SCHOOL A 


20.1^ 


12.8 


53.8 


5.1 


7.7 


87.2 


At 

SCHOOL B 


55. sr 


8.8 


26.5 


6.0 


. 8.8 


91.2 


TOTAL 

Both school a 


37.0 


11.0 


-41.1 


k 

2.7 • 




89.0 
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Tmbl« 3 ^ 
P«rc«nt of Stsff Choi cm 
* Most Apprqprlsts Prof M8.1on«1 
* > To Xnt^rvsns with Physicsl Abuss ' 



\ A • . '. ' ' 


^ — ^ — 

^ School 
1 


Most Approprlsts Profssslonsl 


Total within 
Counssl Ino 
DSDsrtMdnt 


Crisis 
Counsslor 


Qsnsr si 

Counsslor 


' Either 
Counsslor 


Anothsr 
School 
Profssslormi 


NOT 

Appropr^ls.^s 

For,, 
Xntsrvsntipn 

• In Schoot 


At 

SCHOOL A 


42.9 


14.3 


31.0 


9.5 


2.4 


88.1 


SCHOOL /B 


f 
\ 

52.0 


9.9 


. 26.5 


iiTs 

< 


^ ^ . ■ 
2.9 


85. 3 


•c-^^ • " 

TOTAL 

Both Schools 


47.4, 


10.5 


;26.9 


10.5 


2.6 ^ 


86.8 ; 



TsblS 4 
Psrcsnt of Staff Choi ess 
Most Approprlats Profssslonal 
To Intsrvsrvs with Svxual Abuss 

✓ . ■ ■ - - ' 









; Most Approprlats 


Profsssionsi 






School 


"^Crji 
Counii 


lis 

11 or 


Q*nsr»1 

Counsslor 


^ . 

either 
Counssi or 

s 


Andthsr 
School 
Profsssional 


Not 
Approprlsts 

For 

Xntsrvontlon 

In School 


Total within 
Counssl ino 
OapartMnt 


At 

SCHOOL A 


47 


.6 


7.1 


31.0 


9.5 




85.7 


At 

SCHOOL B 


59 


.4 


6.3 


18.8 


9.4 . 


6.3 


84.4 


TOTAL 

Both Schools 


52 


• 

.7 


6.8 


25.7 


9.5 


5.4 


85.1 


y 
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Tabis 5 
Pmrcmnt of Staff ChotcM 
tlost Approprlats ProfMSlonal 



• 


Host Approprlata. Prof •sslonal 






School 


J* . . 

CrIslV 
Counsalbr 


Qaharal 

Counsalor 


Elthar 
c6unsa<1or 


Anothar 
School . 
Profasslonal 


Not-^ 
. Appropr lata 
For 

Intarvantlon 

in scnooi 


Total within 

Counaal ing 
" Dapartmant 


At 

SCHOOL A 


50.0 

• 




26.2 


, ,11.9 


4.8 


83.3 • , 


At 

SCHOOL B 


. 81. 3^ 


3.1 


12.5 


3.1 . 


Q.O 


96.9 


TOTAL 

Both Schools 


/ 63.5 


5:4 


20.3 


8. 1 


•2.7 


89.2 



Tabla 6 , 
Parcant of Staff Cholcaa 
Moaf Approprfata Profaaaldrial^.. 
To Intarvana. ilth Alcohol Abuaa 



.■ . * ■ .*■ 

. ' ■ 'A.' 




■ * ? '. 


liioat >pprbpr1atjb 


Profaaalonal 




4 




% 

School - 


Cr^lrf 
Cbuhial'or 


)S9naral 
Coiinaalpr 


■ . « 

Elthar 
Counaal or 


Anothar 
School 
Profaaalonal 


Not 
Approprlata 

. For 
Intarvantlon 

In School 


Total within 
COMnaal Ino 
Dapartmant 




• At; . 
, SCHOOL A. 


48^.0.. 


.10.0 


.'. ■*■ ' . • 
37.5' 


* ■■■ /' • 
• 5.0 . 


■ ■2.5'; 


92.5 














I ? •• •* ' \. . « 






At' ■ 
• .SCHOOL. B 




"6.5 


32.3 






93.5 




■ • * .-^ • # • > . . 

TOTAL* y 
: Both Schools 


' ■ : 4$JT / 




35.2- 


■•4,.2 ■■• . 




93.0 

^— — r 




♦ : ' • . ■ .' 


■ • « 


















' ' . ' ' 






I - .' ■ ■ ■■ ■ 
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fabl« 7 
Pmrcmnt of Staff Choic«» 
Most Appr opr 1 ato Prof •«» i on« 1 
To intorvoM with OtK»r Substanco Abusa 







Most Approprlata 


Profasslonal 






School 


Crisis 
Counsal or 


Qanaral 

Counsal or 


Elthar 
Counsal or 


. Anothar 
School 
f^rofass^onal 


Not 

Appr opr fatja 

For 

Intarvantlon 

In School 


Total within 

Counsal ing 
' Oapartroant 


At 

SCHOOL A 


48.8 


9.8 


34.1 


"•"'4.9 


2.4 


92.7 


At 

SCHOOL B 


62.5 ^ 


3.1 


25.0 


6.3 


3.1 


90.6 


TOTAL 

Both Sch^ls 


54.8 


6.8 


. 30.1 


5.5 


2.7 


*9i.8 



Tabla 8 
Parcant of Staff Cholcas 
Most Ajppropr lata Profasslonal 
To tntarvana with Pragnaney 



Most Apprpprlata Profasslonal 



. School 


Crisis 
Counsa.lor 


Ganaral 

Counsal or 


Eithar 
Counsal or 


■ Anothar 
School 
Profasslonal 


Not 
Approprlaita 

For. ' 
Intarvantlon 

In^ School 


Total within 
Counsal Ing 
Dapartftant 


■ ( ■' 

At 

SCHOOL A 


39.0 


4.9 


46.3 


4.9 


4.9 


90.2 


At . 

SCHOOL B 


SC.CT 


9.4 


28.1 


6.3 


6.3 


87.5 


TOTAL 

poth Schools 


43.8 


6.8 


38.4 


5.5 


5.5 


89.0 
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table 9 

pfTcent of Staff Choices ^ 
- Moat Appropriate Professional 
To Intervene with Sexually Transmitted Olseaaes 



Most Appropriate Professional 



School 


Crisis 
Counselor 


General 
Counsel or 


f Ither 
Counsel or 


Another 
School 
Prof^esslonai 


Not ^' 
For 

Intervention 

In school 


Total within 
Counsel 1 HQ 
Department 

1 

■ » 


At 

SCHOOL A 


42.5 


5.0 


35.0 ' 


5.0 


12-5 


82.5 


At 

SCHOOL B 


46.9 


3.1 


18.8 


' 21.9 


9.4 


68 .8 


TOTAL 

Both Schools 


44.4 


4.2 


27.8 


12.5 


11.1 


76.4 



Table 10 
Percent of Staff Cholcea^ 
Most Appropriate Profeaslonal 
ToJ Intervene with School Attendance 



Most Appropriate Professional 



School 


Crisis 

Counselor 

> ■ 


General 

Counsel or 


Either 
Counsel or 


Another 
school 
Professional 


Not 
Appropriate 

For 

Intervention 

In School 


Total within 
Counsel ing 
Department 


At 

SCHOOL A^ ^ 


2.6 


'i 

47.4 ^ 


23.7 


26.3 


0.0 


73.7 


At 

SCHOOL B 


3.0 ■ 


21.2 


12.1 


63.6 


. 0.0 


36.4 


TOTAL 

Both Schools 


2.8 


35.2 


18.3 


43.7 


0.0 


56. S. i 



T|bl« 11 
Distribution of Cbbm 
Ptrctnt by Sourct of Rtftrral 





Sourc* Of Btferral 








School >' 


iMchtr 


Qwitral 

Counnlor 


Pt«r 


Stlf 


AdnlnlBtpator 


Partnt 


Citrgy 


Spflcltti 

Education 


SacratarV 


Toti 


SCHOOL A 


0.5 


6.7 


8.8 


58.0 


20.7 


4.1 


0.0 


1.0 


0.0 


Id 


It 

SCHOOL B 


17.6 


8.7 


7.7 


35.5 


14.2 


8.8 


1.1 


4.9 


0.5 


101 


TOTAL 


, 8.8 


7.7 


8.2 


47.1 


17.6 


6.8' 


0.5 


2.9 


0,3 


10! 



Tab1« 12 
Distribution of CasM 
PTcmnt by Typ* of Service 



Typ« of SsrviM 



Individual 
Counseling 



Information 



OTAL 
loth Schools 




EKLC 



